Policy Improvement for POMDPs Using Normalized Importance Sampling

We present a new method for estimating the expected return of a POMDP from experi­ ence. The estimator does not assume any knowledge of the POMDP, can estimate the returns for finite state controllers, allows ex­ perience to be gathered from arbitrary se­ quences of policies, and estimates the return for any new policy. We motivate the estima­ tor from function-approximation and impor­ tance sampling points-of-view and derive its bias and variance. Although the estimator is biased, it has low variance and the bias is of­ ten irrelevant when the estimator is used for pair-wise comparisons. We conclude by ex­ tending the estimator to policies with mem­ ory and compare its performance in a greedy search algorithm to the REINFORCE algo­ rithm showing an order of magnitude reduc­ tion in the number of trials required.